First report of lyrics dataset¶

This first report will an exploration of the lyrics dataset and it will contain:

1. Input data description and overview
2. Data processing steps and methods
3. Data exploration
4. Next steps

After preprocessing the data to contain useful features, I will use analytical approach to find some dependencies or correlations to be able create a meaningful model based on my findings in the next report.

Input data¶

The lyrics dataset contains popular songs with lyrics. There are six columns: id, song name, year of publishing, interpret name, assigned genre and the lyrics. Some rows are music composition without lyrics. There are hyphens instead of spaces in names of songs and interprets. Only one of these columns is numeric - the year. The other columns are strings and will be used as categorical variables.

In this section, we will show what the data looks like and try to understand its basics before further processing.

This is what the dataframe looks like:

The dataset contains 362237 songs, but only 266557 of them contain any lyrics. This means that it is quite big data (it is one of the largest datasets), especially after featurization of the text. Therefore, I will be doing basic analysis using all samples, but some analyses and modelling will be done only using the songs containing lyrics.

Next, you can see a few charts showing the distribution of values in individual columns. For better readability, I have replaced "-" in song and interpreter names with spaces and capitalized first letters.

We can see that most songs were published in 2006 or 2007, so we should be aware that the results reflect rather just the specific culture of those times.

The table also suggests that there might be some data quality issues regarding the release year since there are 10 songs with the year 2038 and are also a few from a distant past.

The dataset is quite biased with its genre representation, so the results cannot be easily generalized to all music. We also have 29814 songs where the genre is not available and 23683 songs where the genre is labeled as "Other".

Number of unique interpreters: 18231

The representation of interpreters is quite good, there are 18231 unique names. However there are a few that have a lot of songs, but 4514 only have one song and 1388 of them only have two songs.

Data processing¶

We have already made the first processing step of replacing hyphens with spaces and capitalizing first results. Now we need to take care of data quality issues and then featurize the song lyrics.

1. Data quality¶

We have already discovered a few data quality issues. Some of the most frequent data quality issues are null values, duplicates, inconsistencies or outliers. I have found no duplicate values in the dataset. So first, we will have a look at columns with null values.

song_name          2
year               0
interpreter        0
genre              0
lyrics         95680
dtype: int64

We can see that these are mostly rows with missing lyrics. Since these columns might not be very useful for further analyses or modelling, but we want to analyze word counts first, so we will just replace them with an empty string.. We can also see that there are two song names missing which we can see in the table above. Those seem to be mistakes so we can drop them.

We can notice from this table that some interpreters can sometimes collaborate with others so we need to be careful that they might also be present in other interpreter names.

Next, we will have a look at rows with suspicious release years.

These rows also seem to be errors. Since we have enough samples, we can also exclude these rows from the dataset (most of these rows will be dropped with null values anyway).

This table revealed another data quality issue - abbreviations like let's or can't are marked the same way as if there were spaces instead of apostrophes. However, this should not be much of a problem since it is consistent between interpreters and song name so we will leave it as is.

Looking at the data also reveals that not all songs are in English. For example the following interpreter sings in German. We will need to take this into consideration when creating a model since creating a single model determining interpreter or genre out of lyrics will be greatly influenced by the language of the song. However, we will not adjust the data for this yet.

There are also inconsistencies in song lyrics and the style in which they are written. Some contain spelling mistakes, but there is not much we can do about that. We could see in the above tables that some lyrics contain parts like "Verse 1:" or "[Chorus]". This would be useful if it was everywhere, but since it is not, it would be better to remove them. We will remove all parts in [] brackets and add "Chorus" and "Verse" to stopwords used when making a model.

2. Featurization¶

Since words cannot be used for modelling as they are, we will need to create some features to be able to analyze them further. We will come back again to featurization in modelling stage since right now I cannot be sure which features we will need. But for exploration, I will create the following features:

• Word count: the total number of words in the lyrics
• Unique word count: the number of unique words in the lyrics
• Average word length: the average length of the words in the lyrics
• Word frequencies (TF-IDF featurization will be used in modelling, however, it is not so good for exploration purposes)

The following boxplot shows the distribution of wordcount and unique word count among genres. Some outliers have been hidden since they would make the chart difficult to read.

This boxplot seems to contain a lot of useful information. For example, we can see that there are genres that are very rarely without lyrics as well as genres that very often have no lyrics. Above all, songs labeled as "Other" rarely have any lyrics. Jazz and Electronic have at least half of their songs with no lyrics. Hip-Hop also has many songs with no lyrics, however, on average, it has the highest amount of words as well as unique words. Although there are many words in pop, there are not many unique ones.

Not Available will probably not be very useful since it just seems to be an average case.

Word length does not differ very much between genres. Generally Metal has the longest words and also the largest variance. One explanation that comes to my mind is that it might often be in a different language, but we can come back to this hypothesis in modelling stage. The word length is mostly around 4.

This is the frequency of individual words in the lyrics, after some basic stopwords were removed. The smaller stopwords version was used with some modifications, since in songs there can be quite a lot of meaning in otherwise meaningless words.

<Axes: >

We can see that term frequency does not differ very much between genres. The chart also contains some spanish words like "que", "de" and "la".

Data exploration¶

Exploration tips:  genres and interprets with most songs  multigenre interprets  distribution of songs features (word count, number of unique words etc.)  song covers and remakes (same name and lyrics, possibly with small differences)  typical words or patterns for various genres and for various interprets

Next steps¶

Based on the results, in my next report, I would like to dedicate space to the following topics:

• creating a representative sample to be able to process data faster
• separating songs based on their language (either by using a library or by unsupervised clustering): this could significantly improve performance of any other model
• prediction of song genre/interpreter/year according to lyrics